Rethinking the Evaluation of Unbiased Scene Graph Generation

Since the severe imbalanced predicate distributions in common subject-object relations, current Scene Graph Generation (SGG) methods tend to predict frequent predicate categories and fail to recognize rare ones. To improve the robustness of SGG models on different predicate categories, recent research has focused on unbiased SGG and adopted mean Recall@K (mR@K) as the main evaluation metric. However, we discovered two overlooked issues about this de facto standard metric mR@K, which makes current unbiased SGG evaluation vulnerable and unfair: 1) mR@K neglects the correlations among predicates and unintentionally breaks category independence when ranking all the triplet predictions together regardless of the predicate categories, leading to the performance of some predicates being underestimated. 2) mR@K neglects the compositional diversity of different predicates and assigns excessively high weights to some oversimple category samples with limited composable relation triplet types. It totally conflicts with the goal of SGG task which encourages models to detect more types of visual relationship triplets. In addition, we investigate the under-explored correlation between objects and predicates, which can serve as a simple but strong baseline for unbiased SGG. In this paper, we refine mR@K and propose two complementary evaluation metrics for unbiased SGG: Independent Mean Recall (IMR) and weighted IMR (wIMR). These two metrics are designed by considering the category independence and diversity of composable relation triplets, respectively. We compare the proposed metrics with the de facto standard metrics through extensive experiments and discuss the solutions to evaluate unbiased SGG in a more trustworthy way

Uncertainty Quantification of Collaborative Detection for Self-Driving

Sharing information between connected and autonomous vehicles (CAVs) fundamentally improves the performance of collaborative object detection for self-driving. However, CAVs still have uncertainties on object detection due to practical challenges, which will affect the later modules in self-driving such as planning and control. Hence, uncertainty quantification is crucial for safety-critical systems such as CAVs. Our work is the first to estimate the uncertainty of collaborative object detection. We propose a novel uncertainty quantification method, called Double-M Quantification, which tailors a moving block bootstrap (MBB) algorithm with direct modeling of the multivariant Gaussian distribution of each corner of the bounding box. Our method captures both the epistemic uncertainty and aleatoric uncertainty with one inference pass based on the offline Double-M training process. And it can be used with different collaborative object detectors. Through experiments on the comprehensive collaborative perception dataset, we show that our Double-M method achieves more than 4X improvement on uncertainty score and more than 3% accuracy improvement, compared with the state-of-the-art uncertainty quantification methods. Our code is public on https://coperception.github.io/double-m-quantification.Comment: 6 pages, 3 figure

Signal processing and generation of bioactive nitric oxide in a model prototissue

The design and construction of synthetic prototissues from integrated assemblies of artificial protocells is an important challenge for synthetic biology and bioengineering. Here we spatially segregate chemically communicating populations of enzyme-decorated phospholipid-enveloped polymer/DNA coacervate protocells in hydrogel modules to construct a tubular prototissue-like vessel capable of modulating the output of bioactive nitric oxide (NO). By decorating the protocells with glucose oxidase, horseradish peroxidase or catalase and arranging different modules concentrically, a glucose/hydroxyurea dual input leads to logic-gate signal processing under reaction-diffusion conditions, which results in a distinct NO output in the internal lumen of the model prototissue. The NO output is exploited to inhibit platelet activation and blood clot formation in samples of plasma and whole blood located in the internal channel of the device, thereby demonstrating proof-of-concept use of the prototissue-like vessel for anticoagulation applications. Our results highlight opportunities for the development of spatially organized synthetic prototissue modules from assemblages of artificial protocells and provide a step towards the organization of biochemical processes in integrated micro-compartmentalized media, micro-reactor technology and soft functional materials

Prediction of phosphotyrosine signaling networks using a scoring matrix-assisted ligand identification approach

Systematic identification of binding partners for modular domains such as Src homology 2 (SH2) is important for understanding the biological function of the corresponding SH2 proteins. We have developed a worldwide web-accessible computer program dubbed SMALI for scoring matrix-assisted ligand identification for SH2 domains and other signaling modules. The current version of SMALI harbors 76 unique scoring matrices for SH2 domains derived from screening oriented peptide array libraries. These scoring matrices are used to search a protein database for short peptides preferred by an SH2 domain. An experimentally determined cut-off value is used to normalize an SMALI score, therefore allowing for direct comparison in peptide-binding potential for different SH2 domains. SMALI employs distinct scoring matrices from Scansite, a popular motif-scanning program. Moreover, SMALI contains built-in filters for phosphoproteins, Gene Ontology (GO) correlation and colocalization of subject and query proteins. Compared to Scansite, SMALI exhibited improved accuracy in identifying binding peptides for SH2 domains. Applying SMALI to a group of SH2 domains identified hundreds of interactions that overlap significantly with known networks mediated by the corresponding SH2 proteins, suggesting SMALI is a useful tool for facile identification of signaling networks mediated by modular domains that recognize short linear peptide motifs

Failure Mechanism of Bolts and Countermeasures in Swelling Soft Rock Support

The effect of conventional bolt support is not ideal due to the large deformation character of soft rock. As an innovative bolt, constant resistance large deformation (CRLD) bolt has been successfully applied to swelling soft rock engineering, but the reinforcement mechanism is not yet clear. To investigate the interaction mechanism between bolt and surrounding rock, Nanshan Coal Mine of China was selected as the engineering background. The plastic zone of surrounding rock mass and the axial force of three bolts were obtained by theoretical analysis and FLAC3D numerical simulation. Failure processes of conventional pretension bolts in soft rock were reproduced, and the interaction between CRLD bolt and soft rock was investigated in comparison. The results indicate that: (1) The fracture zone of surrounding rock exceeds the anchorage range of the low pretension bolt, the bolt slides with surrounding rock and finally fails. (2) The fracture zone of surrounding rock does not exceed the anchorage range of the high pretension bolt. However, with the accumulation of deformation energy, stress concentration makes the bolt break. (3) CRLD bolt can effectively absorb the deformation energy released by soft rock and maintain constant support resistance. The conclusions obtained in this study provide significant references in the selection of bolts in soft rock engineering

MMBench: Is Your Multi-modal Model an All-around Player?

Large vision-language models have recently achieved remarkable progress, exhibiting great perception and reasoning abilities concerning visual information. However, how to effectively evaluate these large vision-language models remains a major obstacle, hindering future model development. Traditional benchmarks like VQAv2 or COCO Caption provide quantitative performance measurements but suffer from a lack of fine-grained ability assessment and non-robust evaluation metrics. Recent subjective benchmarks, such as OwlEval, offer comprehensive evaluations of a model's abilities by incorporating human labor, but they are not scalable and display significant bias. In response to these challenges, we propose MMBench, a novel multi-modality benchmark. MMBench methodically develops a comprehensive evaluation pipeline, primarily comprised of two elements. The first element is a meticulously curated dataset that surpasses existing similar benchmarks in terms of the number and variety of evaluation questions and abilities. The second element introduces a novel CircularEval strategy and incorporates the use of ChatGPT. This implementation is designed to convert free-form predictions into pre-defined choices, thereby facilitating a more robust evaluation of the model's predictions. MMBench is a systematically-designed objective benchmark for robustly evaluating the various abilities of vision-language models. We hope MMBench will assist the research community in better evaluating their models and encourage future advancements in this domain. Project page: https://opencompass.org.cn/mmbench